Method for monitoring temperature-controlled units in a store

ABSTRACT

One variation of a method for monitoring cooling units in a store includes: at a robotic system, during a first scan routine, autonomously navigating toward a cooling unit in the store, recording a color image of the cooling unit, and scanning a set of temperatures within the cooling unit; identifying a set of products stocked in the cooling unit based on features detected in the color image; mapping the set of temperatures to the set of products at a first time during the first scan routine based on positions of products in the set of products identified in the color image; and generating a record of temperatures of the set of products stocked in the cooling unit at the first time.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/615,809, filed on 10 Jan. 2018, which is incorporatedin its entirety by this reference.

This application is related to U.S. patent application Ser. No.15/600,527, filed on 19 May 2017, and to U.S. patent application Ser.No. 15/347,689, filed on 9 Nov. 2016, each of which is incorporated inits entirety by this reference.

TECHNICAL FIELD

This invention relates generally to the field of inventory managementand more specifically to a new and useful method for monitoringtemperature-controlled units in a store in the field of inventorymanagement.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a flowchart representation of a method;

FIG. 2 is a schematic representation of one variation of the method; and

FIG. 3 is a flowchart representation of one variation of the method.

DESCRIPTION OF THE EMBODIMENTS

The following description of embodiments of the invention is notintended to limit the invention to these embodiments but rather toenable a person skilled in the art to make and use this invention.Variations, configurations, implementations, example implementations,and examples described herein are optional and are not exclusive to thevariations, configurations, implementations, example implementations,and examples they describe. The invention described herein can includeany and all permutations of these variations, configurations,implementations, example implementations, and examples.

1. Method

As shown in FIG. 1, a method S100 for monitoring temperature-controlledunits in a store includes, at a robotic system, during a first scanroutine: autonomously navigating toward a cooling unit in the store inBlock S110; recording a color image of the cooling unit in Block S120;and scanning a set of temperatures within the cooling unit in BlockS130. The method S100 also includes: identifying a set of productsstocked in the cooling unit based on features detected in the colorimage in Block S140; mapping the set of temperatures to the set ofproducts at a first time during the first scan routine based onpositions of products in the set of products identified in the colorimage in Block S142; and generating a record of temperatures of the setof products stocked in the cooling unit at the first time in Block S170.

One variation of the method S100 includes, at the robotic system, duringa first scan routine: autonomously navigating toward atemperature-controlled unit in the store in Block S110; and recording acolor image and a thermal image of the temperature-controlled unit inBlock S120. This variation of the method S100 also includes: identifyinga set of products stocked in the temperature-controlled unit based onfeatures detected in the color image in Block S140; for each product inthe set of products stocked in the temperature-controlled unit,retrieving a stored emissivity of a surface of the product from adatabase based on an identity of the first product in Block S150,selecting a thermal pixel in the thermal image depicting the product inBlock S130, and, calculating a temperature of the product at the firsttime based on a thermal value stored in the thermal pixel and theemissivity of the surface of the product in Block S148; and generating arecord of temperatures of products, in the set of products, at the firsttime in Block S170. In this variation, the method S100 can furtherinclude: at the robotic system during the first scan routine, recordinga depth image of the open cooling unit in Block S132 and recording anambient temperature, an ambient humidity, and an ambient light levelproximal the robotic system in Block S132; and, for each product in theset of products, calculating a distance from the robotic system to theproduct based on the depth image in Block S154 and passing theemissivity of the surface of the product, the thermal value stored inthe thermal pixel, the distance, the ambient temperature, the ambienthumidity, and the ambient light level into a physics model to calculatethe temperature of the product at the first time in Block S156.

Another variation of the method S100 includes: at a robotic system,during a first scan routine: autonomously navigating toward atemperature-controlled unit in the store in Block S110; recording acolor image of the temperature-controlled unit in Block S120; andbroadcasting a query for temperature values from a set oftemperature-enabled transmitters coupled to the temperature-controlledunit in Block S160; and recording a set of temperature valuestransmitted by the set of temperature-enabled transmitters in BlockS130. This variation of the method S100 also includes: identifying a setof products stocked in the temperature-controlled unit based on featuresdetected in the color image in Block S140; accessing locations of theset of temperature-enabled transmitters coupled to thetemperature-controlled unit; mapping the set of temperature valuesreceived from the set of temperature-enabled transmitters to the set ofproducts at the first time in Block S142 based on locations of the setof temperature-enabled transmitters in the temperature-controlled unitand locations of the set of products identified in the color image; andgenerating a record of temperatures of the set of products stocked inthe temperature-controlled unit at the first time in Block S170.

2. Applications

Generally, the method S100 can be executed by a robotic system and by acomputer system (hereinafter the “system”): to autonomously collectproduct identification, temperature, humidity, and/or other relevantdata from temperature-controlled units (e.g., open refrigerators,enclosed refrigerators, freezers, heated and cooled food displays, etc.)within a facility (e.g., a grocery store); and to transform these datainto temperatures of specific products stocked in thesetemperature-controlled units, temperatures of product slots in thesetemperature-controlled units, and/or temperatures of thesetemperature-controlled units more generally. Based on derivedtemperatures of such products, the system can selectively alertassociates of the facility to remove and discard specific products thathave been exposed to temperatures outside of predefined temperatureranges. Similarly, based on derived temperatures of product slots, thesystem can selectively prompt associates of the facility to movespecific products to different product slots better suited to assignedtemperature storage ranges of these products. Furthermore, based onderived temperatures of such temperature-controlled units (e.g., averagetemperature, temperature gradient, spatial or temporal temperaturevariance), the system can selectively alert associates of the facilitywhen these temperature-controlled units are malfunctioning or set atincorrect temperatures for products stocked in these units.

In particular, temperatures of temperature-controlled units and productsstored therein may be audited manually by associates of a store.However, such audits may be time-consuming and therefore performedintermittently or rarely such that detection of malfunctioningtemperature-controlled units is significantly delayed such as by hoursor by days and only following a customer complaint. Intermittenttemperature data of these temperature-controlled units may further yielduncertainty regarding both when a temperature-controlled unit began tomalfunction and actual temperature exposures of products stored in thistemperature-controlled unit. Therefore, to compensate for greater riskof food spoilage given such temperature exposure uncertainty, the storemay discard the entire contents of the malfunctioningtemperature-controlled unit.

2.1 Open Temperature-Controlled Units

Conversely, a robotic system deployed in the store can execute Blocks ofthe method S100 to: autonomously navigate throughout the store (e.g.,dynamically based on patron traffic or along a predefined sequence ofwaypoints); and record temperature data of temperature-controlled unitsat known locations throughout the store. For example, at each waypointassociated with an open temperature-controlled unit (e.g., an openrefrigerator) within the store, the robotic system can: orient towardthe adjacent temperature-controlled unit; record a thermal image, acolor image, and/or a depth image of the temperature-controlled unitthrough a thermographic camera, a color camera, a depth sensorintegrated into the robotic system; record local ambient data via othersensors integrated into the robotic system; and upload these data to aremote computer system via a wireless local area network. The remotecomputer system can then: implement methods and techniques described inU.S. patent application Ser. No. 15/600,527—which is incorporated in itsentirety by this reference—to identify products in thetemperature-controlled unit based on features extracted from the colorimage; retrieve known thermal coefficients for these products (e.g.,average thermal emissivity values of product packaging of theseproducts); and then calculate corrected temperatures of these productsbased on thermal values stored in pixels in the thermal image depictingthese products identified in the color image, thermal coefficients ofthese products, local ambient data, and distances between the roboticsystem (e.g., the thermographic camera) and these products. Additionallyor alternatively, the remote computer system can implement similarmethods and techniques to calculate corrected temperatures of referencesurfaces in the temperature-controlled unit, such as faces of shelvessupporting these products. The robotic system and remote computer systemcan repeat this process for each other open temperature-controlled unitin the store, record these temperature data in a database or othermemory format, and then selectively serve prompts or notifications toassociates of the store, maintenance staff, etc. based on thesetemperature data.

2.2 Enclosed Temperature-Controlled Units

Additionally or alternatively, open and/or enclosed cooling units (e.g.,refrigerators and freezers with sealed doors) throughout the store canbe outfitted with temperature-enabled transmitters (e.g.,temperature-enabled RFID or NFC tags) configured to broadcast uniqueelectronic identifiers and temperature values when excited by a powersignal broadcast by a reader integrated into the robotic system. Whenoccupying a location (e.g., a predefined waypoint) adjacent atemperature-controlled unit outfitted with such temperature-enabledtransmitters, the robotic system can: record a color image of thetemperature-enabled unit; broadcast an excitation signal toward thetemperature-controlled unit; record unique electronic identifier andtemperature value pairs returned by temperature-enabled transmittersnearby; and upload the color image and unique electronic identifier andtemperature value pairs to the remote computer system. The remotecomputer system can then: reference a lookup table or other databaselinking these unique electronic identifiers to specific cooling units orcooling unit locations throughout the store; derive positions of thesetemperature-enabled transmitters based on positions and orientations ofthe robotic system and characteristics of signals received from thesetemperature-enabled transmitters; or visually detect thesetemperature-enabled transmitters (e.g., constellations of unique opticalfeatures linked to unique identifies broadcast by correspondingtemperature-enabled transmitters) in the color image. The remotecomputer system can then interpolate temperatures of products and/orreference surfaces on the temperature-controlled unit—detected in thecolor image—based on temperature values received from thesetemperature-enabled transmitters and known or derived positions of thesetemperature-enabled transmitters. The remote computer system can repeatthis process for other open or enclosed temperature-controlled unitsoutfitted with such temperature-enabled transmitters throughout thestore in order to monitor temperatures of products stocked in thesetemperature-controlled units.

2.3 Temperature Data Aggregation and Prompts

By regularly executing Blocks of the method S100 to record temperaturedata of temperature-controlled units throughout the store andtemperature data of product stocked therein—such as each time therobotic system passes a temperature-controlled unit during dailyinventory tracking routines or hourly temperature monitoring routines(generally “scan routines”)—the robotic system can amass a timeseries oftemperature-controlled unit and product temperatures over time. Based onthese temperature data, the remote computer system can rapidly detectabnormal mean temperatures, temperature variances, or temperaturegradients, etc. in temperature-controlled units throughout the store andthen prompt store associates to inspect, adjust, or repair thesetemperature-controlled units accordingly, which may enable these storeassociates to address an incorrect temperature setting or malfunction ata temperature-controlled unit before products stored in themalfunctioning unit spoil or are otherwise exposed to temperaturesoutside of a required bound.

The remote computer system can also detect temperature-related trends intemperature-controlled units in the store, predict probability of futurefailure of a particular temperature-controlled unit based on atemperature-related trend of the particular temperature-controlled unit(e.g., a trend in temperature versus power consumption, temperaturevariance, mean temperature change over time), and then issue a prompt tostore associates to preemptively inspect, repair, or replace theparticular unit, thereby enabling store associates to preemptivelyaddress future temperature-controlled unit failures.

Furthermore, by generating and storing this timeseries of temperaturedata of temperature-controlled units, product slots, and/or individualproducts in temperature-controlled units throughout the store, theremote computer system can maintain a log of temperature exposures ofperishable goods stocked throughout the store and sold by the store overtime, thereby: enabling the store (or a disease prevention authority orother external regulatory entity) to determine—with greaterconfidence—whether spoilage of a particular perishable good occurred dueto temperature exposure at the store or due to exposure or other factoroutside of the store; and enabling the store to systematically addresssuch temperature-related errors.

Therefore, the robotic system can regularly collect temperature data oftemperature-controlled units throughout a store, such as systematicallymultiple times per day during inventory tracking routines and/ordedicated temperature monitoring routines. The remote computer system(or the robotic system) can process these data in near real-time torapidly or preemptively detect a malfunctioning temperature-controlledunit and to inform associates of the store accordingly. The remotecomputer system can also maintain temperature logs oftemperature-controlled units, product slots, or specific productsthroughout the store over time (e.g., multiple temperature recordingsper day for each temperature-controlled unit over a period of sixmonths) in order to provide longer-term insight into temperatureexposure of perishable goods stocked in these temperature-controlledunits sold by the store.

The method S100 is described below as executed by the roboticsystem—deployed in a grocery store—and by the remote computer system tomonitor cooling units arranged in a grocery store, to track temperaturesof products stocked in these cooling units over time, and to selectivelyserve prompts related to temperatures of these products and function ofthese cooling units. However, the robotic system and the remote computersystem can implement similar methods and techniques to monitor foodbars, heated cabinets, heated displays, and/or temperature-controlledunits of other types located within a grocery store, cafeteria,restaurant, or other type of facility.

Furthermore, the method S100 is described herein as including: Blocksexecuted locally by the robotic system to collect stock- andtemperature-related data throughout a store; and Blocks executedremotely by a remote computer system (e.g., a remote server, a computernetwork, a computer system within the store and remote or separate fromthe robotic system). However, Blocks of the method S100 can be executedby one or more robotic systems placed in a store (or warehouse, etc.),by a local or remote computer system, and/or by any other computersystem. For example, the robotic system can also upload scan data (e.g.,thermal images, depth maps, color images) to a remote computersystem—such as over a cellular network or local area network—for remoteprocessing and by the remote computer system. However, the roboticsystem can additionally or alternatively execute Blocks of the methodS100 to locally process stock- and temperature-related data, such as inreal-time during an inventory tracking routine or dedicated temperaturemonitoring routine.

3. Robotic System

Generally, a robotic system can define a network-enabled mobile robotthat can autonomously traverse a store, capture images of shelves withinthe store, and upload those images to a remote computer system foranalysis, as shown in FIGS. 2 and 3. In particular, the robotic systemexecutes Blocks S110, S120, S130, S160, and S130, etc. of the methodS100 to: autonomously navigate throughout a store; capture color,thermographic, and/or depth images of shelving structures andtemperature-controlled units throughout the store; querytemperature-enabled transmitters distributed throughouttemperature-controlled units in the store for temperature values; and/orcollect ambient data throughout the store. The robotic system can thenupload these (timestamped and georeferenced) scan data, temperaturedata, and ambient data to the remote computer system, such as over acellular network or local area network in real-time during a scanroutine or upon conclusion of the scan routine. The remote computersystem can then execute Blocks of the method S100 described below toremotely process these data.

In one implementation, the robotic system defines an autonomous imagingvehicle including: a base; a drive system (e.g., a pair of two drivenwheels and two swiveling castors) arranged in the base; a power supply(e.g., an electric battery); a set of depth sensors (e.g., fore and aftscanning LIDAR systems); a processor that transforms data collected bythe depth sensors into two- or three-dimensional maps of a space aroundthe robotic system; a mast extending vertically from the base; a set ofcameras (e.g., color cameras, thermographic cameras, and/or depthsensors) arranged on the mast; a geospatial position sensor (e.g., a GPSsensor); and a wireless communication module that downloads a sequenceof waypoints and a master map of a store from a remote computer system(e.g., a remote server) and that uploads color, thermographic, and/ordepth images captured by sensors in the robotic system to the remotecomputer system, as shown in FIG. 2.

In this implementation, the robotic system can include cameras mountedstatically to the mast, such as two vertically offset sets of color,thermographic, and depth sensors on a left side of the mast and twovertically offset sets of color, thermographic, and depth sensors on aright side of mast, as shown in FIG. 2. The robotic system canadditionally or alternatively include articulable cameras, such as: oneset of cameras on the left side of the mast and supported by a firstvertical scanning actuator; and one set of cameras on the right side ofthe mast and supported by a second vertical scanning actuator. Forexample, each color camera can be configured to output a color (e.g.,RGB) photographic image. Each depth sensor can include a LIDAR sensor,RADAR sensor, 3D stereoscopic camera, or structured-light 3D scanner,etc. configured to output one- or multi-dimensional depth images,wherein each pixel in a depth image contains a distance value from thedepth sensor at the robotic system to an unique point on an adjacentsurface (e.g., a surface of a product stored on a shelf in an adjacentcooling unit) in the field of view of the depth sensor. Eachthermographic camera (or infrared camera, thermal imaging camera, etc.)can be configured to output 2D thermal images, wherein each pixel in athermal image contains a thermal radiation value of a unique point on anadjacent surface (e.g., a surface of a product stored on a shelf in anadjacent cooling unit) in the field of view of the thermographic camera.The remote computer system (or the robotic system) can interpret theradiation value in a pixel in the thermal image as a temperature of thesurface represented by this pixel, such as by correcting this radiationvalue based on distance between the robotic system the surface, ambienttemperature, ambient humidity, and a known or predicted thermalcoefficient (e.g., thermal emissivity) of the surface, as describedbelow.

The robotic system can therefore further include: a temperature sensorconfigured to measure ambient temperature proximal the robotic system; ahumidity sensor configured to measure ambient humidity proximal therobotic system (which may be correlated to humidity in the cooling unit,such as as a function of distance from the humidity sensor in therobotic system to the cooling unit); and/or an ambient light sensorconfigured to measure an ambient light level proximal the roboticsystem.

The robotic system can also include a wireless transmitter and receiverthat cooperate to broadcast an interrogation signal and to collect anddiscern inbound wireless signals broadcast from temperature-enabledtransmitters—excited by the interrogation signal—located nearby. Forexample, the robotic system can include: a radio-frequencyidentification (or “RFID”) antenna configured to broadcast aninterrogation signal toward a temperature-controlled unit near therobotic system; and a RFID reader configured to read substantiallyunique electronic identifier (or “UUID”) and temperature value pairstransmitted by RFID tags arranged within the temperature-controlledunit.

In one variation, the robotic system includes multiple RFID antennas.For example, the robotic system can include a first RFID antennaarranged in a first polarization orientation at a first position alongthe mast; and a second RFID antenna arranged in a second polarizationorientation at a second position along the mast. In this example, thesecond polarization orientation can be angularly offset from the firstpolarization orientation by a known angle (e.g., 90°) about a horizontalaxis of the robotic system; and the second position can be verticallyoffset above the first position by a known distance (e.g., 50centimeters). During a scan routine, the robotic system can: triggerboth the first and second RFID antennas to broadcast interrogationsignals; collect RF signals through both the first and second RFIDantennas; and compile these RF signals and related metadata into a 2D or3D map of locations of RFID tags—from which these RF signalsoriginated—based on known linear offsets between the first and secondantennas. Furthermore, a particular RFID tag parallel to the plane ofpropagation of an interrogation signal broadcast by the first antennamay not return an RF signal to the robotic system; however, because thesecond antenna is angularly offset from the first antenna, theparticular RFID tag may be necessarily non-parallel to the plane ofpropagation of an interrogation signal broadcast by the second antenna.Therefore, the robotic system can include two (or more) non-parallelRFID antennas in order to enable collection of RF signals from a greaterproportion of nearby RFID tags, including RFID tags that may be obscuredfrom one RFID antenna in the set, such as described in U.S. patentapplication Ser. No. 15/947,757.

However, the robotic system can define any other form and can includeany other subsystems, elements, or sensors to enable the robotic systemto autonomously navigate throughout a store to and record color,distance, thermal, and/or ambient data of shelving structures andcooling units throughout the store.

Furthermore, multiple instances of the robotic system can be placed in asingle store and configured to cooperate to image shelving structuresand temperature-controlled units within the store. For example, tworobotic systems can be placed in a large, single-floor retail store andcan cooperate to collect images of all shelving structures andtemperature-controlled units in the store within a limited duration oftime (e.g., within one hour). In another example, one robotic system canbe placed on each floor of a multi-floor store, and each robotic systemcan collect images of shelves on its corresponding floor. The remotecomputer system can then aggregate images captured by multiple roboticsystems placed in one store to generate a graph, map, table, and/or tasklist of properly- and improperly-stocked slots within the store.

4. Hierarchy and Terms

A “product facing” is referred to herein as a side of a product (e.g.,of a particular SKU or other product identifier) designated for a slot.A “planogram” is referred to herein as a graphical representation ofmultiple product facings across each of multiple shelving structureswithin a store (e.g., across an entire store). Product identification,placement, and orientation data recorded visually in a planogram can bealso be recorded in a corresponding textual product placementspreadsheet, slot index, or other store database (hereinafter a “productplacement database”).

A “slot” is referred to herein as a section of a shelf designated foroccupation by one product facing, such as including a row of one or moreunits of a product. A “shelf” is referred herein as one lateral surface(e.g., one four-foot-wide horizontal surface) spanning one or moreslots. A “shelving segment” is referred to herein as one column of ashelving structure, including one or more shelves. A “shelvingstructure” is referred to herein as a row of one or more shelvingsegments. An “aisle” is referred to herein as a thoroughfare between twoopposing shelving structures. A “store” is referred to herein as a(static or mobile) facility containing one or more shelving structuresand one or more aisles.

Furthermore, a temperature-controlled unit is referred to herein as aunit containing one or more slots, shelves, and/or shelving segments andconfigured to store one or more products at a temperature above or belowa nominal ambient indoor air temperature. An enclosedtemperature-controlled unit (e.g., a freezer or refrigeration unit) caninclude a glass or otherwise transparent door that isolates atemperature-controlled interior cavity from an ambient environment.

5. Robotic System Dispatch

Block S110 of the method S100 recites, at the robotic system,autonomously navigating toward a cooling unit in the store during a scancycle. Generally, in Block S110, the autonomous vehicle can autonomouslynavigate throughout the store, such as: during an inventory trackingroutine in which the robotic system's primary function is to recordimages of inventory structures for product tracking and derivation ofthe current stock state of the store; or during a dedicated temperaturemonitoring routine in which the robotic system's primary function is tomonitor temperatures of temperature-controlled units (or productscontained in these temperature-controlled units more specifically).

5.1 Inventory Tracking Routine

In one implementation, the robotic system executes Blocks of the methodS100 while executing an inventory tracking routine within the store. Inthis implementation and as shown in FIG. 3, the remote computer system(e.g., a remote server connected to the robotic system via theInternet): defines a set of waypoints specifying target locations withinthe store at which the robotic system is to navigate and capture imagesof inventory structure throughout the store; and intermittently (e.g.,twice per day) dispatches the robotic system to navigate through thissequence of waypoints and to record images of inventory structuresnearby during an inventory tracking routine. For example, the roboticsystem can be installed within a retail store (or a warehouse, etc.),and the remote computer system can dispatch the robotic system toexecute an inventory tracking routine during store hours, includingnavigating to each waypoint throughout the retail store and collectingdata representative of the stock state of the store in real-time aspatrons move, remove, and occasionally return product on, from, and toinventory structures throughout the store. Alternatively, the remotecomputer system can dispatch the robotic system to execute thisinventory tracking routine outside of store hours, such as every nightbeginning at 1 AM. The robotic system can thus complete an inventorytracking routine before the retail store opens hours later.

In a similar implementation, the remote computer system: dispatches therobotic system to navigate along aisles within the store (e.g., throughthe sequence of predefined waypoints within the store) and to captureimages of products arranged on inventory structures (e.g., shelvingstructures, refrigeration units and other temperature-controlled units,displays, hanging racks, cubbies, etc.) throughout the store during aninventory tracking routine; downloads color images of these inventorystructures recorded by the robotic system; and implements imageprocessing, computer vision, artificial intelligence, deep learning,and/or other methods and techniques to estimate the current stockingstatus of these inventory structures based on products detected in theseimages. The robotic system can additionally or alternatively broadcastradio frequency queries and record RFID data from RFID tags arranged onor integrated into products stocked throughout the store during theinventory tracking routine; and the remote computer system can downloadthese RFID data from the robotic system and detect locations andquantities of products throughout the store based on these data. Theremote computer system can then automatically generate a stocking reportfor the store, such as including slots or other product locations thatare sufficiently stocked, understocked, incorrectly stocked, and/ordisheveled as described in U.S. patent application Ser. No. 15/347,689.

For example, the robotic system can be placed within a retail store (orwarehouse, etc.). The remote computer system can: dispatch the roboticsystem to collect data at waypoints throughout the retail store outsideof store hours, such as every night beginning at 1 AM. The roboticsystem can thus complete a scan routine before the retail store opens afew hours later; process these data to generate a map of current productplacement on shelves in the store and/or to generate a task list ofmisplaced or misoriented products to correct; and then present a mapand/or a task list to associates upon their arrival at the retail storethe next morning before the retail store opens. The remote computersystem can additionally or alternatively dispatch the robotic system toexecute a scan routine during open store hours and can process datareceived from the robotic system substantially in real-time to generatesuch maps and/or task lists in near-real-time.

The remote computer system can therefore maintain, update, anddistribute a set of waypoints to the robotic system, wherein eachwaypoint defines a location within a store at which the robotic systemis to capture one or more images from the integrated thermographic,depth, and/or color cameras. In one implementation, the remote computersystem defines an origin of a two-dimensional Cartesian coordinatesystem for the store at a charging station—for the robotic system—placedin the store, and a waypoint for the store defines a location within thecoordinate system, such as a lateral (“x”) distance and a longitudinal(“y”) distance from the origin. Thus, when executing a waypoint, therobotic system can navigate to (e.g., within three inches of) a (x,y)coordinate of the store as defined in the waypoint. For example, for astore that includes shelving structures with four-foot-wide shelvingsegments and six-foot-wide aisles, the remote computer system can defineone waypoint laterally and longitudinally centered—in a correspondingaisle—between each opposite shelving segment pair. A waypoint can alsodefine a target orientation, such as in the form of a target angle (“∂”)relative to the origin of the store, based on an angular position of anaisle or shelving structure in the coordinate system, as shown in FIG.5. When executing a waypoint, the robotic system can orient to (e.g.,within 1.5° of) the target orientation defined in the waypoint in orderto align a camera to an adjacent shelving structure.

When navigating to a waypoint, the robotic system can scan anenvironment nearby with the depth sensor (e.g., a LIDAR sensor, asdescribed above), compile depth scans into a new map of the roboticsystem's environment, determine its location within the store bycomparing the new map to a master map of the store defining thecoordinate system of the store, and navigate to a position andorientation within the store at which the output of the depth sensoraligns—within a threshold distance and angle—with a region of the mastermap corresponding to the (x,y,∂) location and target orientation definedin the waypoint. A waypoint can also include a geospatial position(e.g., a GPS location), such as in the form of a backup or redundantlocation. For example, when navigating to a waypoint, the robotic systemcan approach the geospatial position defined in the waypoint; oncewithin a threshold distance (e.g., five feet) from the geospatialposition, the remote computer system can navigate to a position andorientation at which the output of the depth sensor aligns—within athreshold distance and angle—with a region of the master mapcorresponding to the (x,y,∂) location and target orientation defined inthe waypoint.

Furthermore, a waypoint can include an address of each camera that is tocapture an image once the robotic system can navigate to the waypoint.For example, for the robotic system that includes a thermographiccamera, a depth camera, and a color camera, the waypoint can include allor a subset of camera addresses [1, 2, 3] corresponding to athermographic camera, a depth camera, and a color camera, respectively.Alternatively, for the robotic system that includes articulable cameras,a waypoint can define an address and arcuate position of each camerathat is to capture an image at the waypoint. Similarly, a waypoint caninclude a specification for: recordation of a thermal image, an ambienthumidity, and/or an ambient temperature; and/or for broadcasting aquerying to temperature-enabled transmitters (e.g., RFID tags) nearby.The robotic system can then record color images, depth maps, thermalimages, ambient humidity data, ambient temperature data, and/ortransmitter identifier and temperature value pairs when occupying awaypoint based on a data specification contained in this waypoint.

In one implementation, before initiating a new inventory trackingroutine, the robotic system can download—from the remote computersystem—a set of waypoints, a preferred order for the waypoints, and amaster map of the store defining the coordinate system of the store.Once the robotic system leaves its dock at the beginning of an inventorytracking routine, the robotic system can repeatedly sample itsintegrated depth sensors (e.g., a LIDAR sensor) and construct a new mapof its environment based on data collected by the depth sensors. Bycomparing the new map to the master map, the robotic system can trackits location within the store throughout the inventory tracking routine.Furthermore, to navigate to a next waypoint, the robotic system canconfirm its achievement of the waypoint—within a threshold distance andangular offset—based on alignment between a region of the master mapcorresponding to the (x,y,∂) location and target orientation defined inthe current waypoint and a current output of the depth sensors, asdescribed above.

Alternatively, the robotic system can execute a waypoint defining a GPSlocation and compass heading and can confirm achievement of the waypointbased on outputs of a GPS sensor and compass sensor within the roboticsystem. However, the robotic system can implement any other methods ortechniques to navigate to a position and orientation within the storewithin a threshold distance and angular offset from a location andtarget orientation defined in a waypoint.

Yet alternatively, during an inventory tracking routine, the roboticsystem can autonomously generate a path throughout the store and executethis path in real-time based on: obstacles (e.g., patrons, spills,inventory structures) detected nearby; priority or weights previouslyassigned to inventory structures, temperature-controlled units, orparticular slots within the store; and/or product sale data from apoint-of-sale system connected to the store and known locations ofproducts in the store, such as defined in a planogram; etc. For example,the computer system can dynamically generate its path throughout thestore during an inventory tracking routine to maximize a value ofinventory structures or particular products imaged by the robotic systemper unit time responsive to dynamic obstacles within the store (e.g.,patrons, spills, shopping carts), such as described in U.S. patentapplication Ser. No. 15/347,689.

Therefore, in this implementation, the robotic system can autonomouslynavigate along a set of aisles within the store during an inventorytracking routine. While autonomously navigating along a particular aislein this set of aisles in the store, the robotic system can: record a setof color images of a set of shelving structures facing the particularaisle; record a thermal image of the particular aisle at a first timeand/or query temperature-enabled transmitters nearby; record ambientdata; and then upload these data—such as in real-time or upon conclusionof the inventory tracking routine—to the remote computer system forprocessing as described below. In this implementation, the roboticsystem can repeat the foregoing process to record similar data imagesalong each inventory structure and temperature-controlled unit in thestore during the inventory tracking routine; and the remote computersystem can similarly process these data to derive a current stockingstate of the entire store.

5.2 Dedicated Temperature Monitoring Routine

Additionally or alternatively, the robotic system can autonomouslynavigate throughout the store and execute Blocks of the method S100 tospecifically track temperatures of temperature-controlled units and/orproducts contained therein. For example, when not executing inventorytracking routines and recharging at a dock located in the store and/orduring high traffic periods in the store, the robotic system canautonomously execute a dedicated temperature monitoring routine,including: sequentially navigating to each temperature-controlled unitin the store; executing Blocks of the method S100 to record colorimages, depth maps, thermal images, depth maps, transmitter identifierand temperature value pairs, and/or ambient data; and then transmittingthese data to the remote computer system for processing.

In one implementation, the remote computer system can access a planogramof the store; scan the planogram to identify each temperature-controlledunit in the store. The remote computer system can then assign prioritiesto scanning these temperature-controlled units: as a function of (e.g.,proportional to) size, age, or magnitude last detected spatial ortemporal temperature variance of these temperature-controlled units; inorder of value of products stocked in these temperature-controlledunits; and/or in order to temperature sensitivity of products stored inthese temperature-controlled units (e.g., highest priority for cookedfoods in a buffet-style hot food display; high priority for dairy goodscontained in an open cooling unit; moderate priority for ice cream; lowpriority for refrigerated dairy goods in an enclosed cooling unit).During a next temperature monitoring routine, the robotic system canautonomously navigate between these temperature-controlled units andscan these temperature-controlled units according to their priority (anda route that minimizes distance or time traveled).

Furthermore, the robotic system can execute temperature monitoringroutines separately from inventory tracking routines in the store.Alternately, the computer system can execute consecutive inventorytracking and temperature monitoring routines. For example, the roboticsystem can navigate off of its dock and initiate an inventory trackingroutine at a scheduled time. Upon completing the inventory trackingroutine, the robotic system can transition to executing a temperaturemonitoring routine until a charge state of the robotic system dropsbelow a low threshold, at which time the robotic system can navigateback to its dock to recharge before a next scheduled inventory trackingroutine.

6. Scan Routine

Block S120 of the method S100 recites recording a color image of thecooling unit; and Block S130 of the method S100 recites scanning a setof temperatures within the cooling unit. Generally, in Block S120, therobotic system can record a color image (e.g., a digital colorphotographic image) of a temperature-controlled unit, which the remotecomputer system (or the robotic system) can then process to identifyproducts currently stocked in the temperature-controlled unit. In BlockS130, the robotic system can collect temperature-related data of thetemperature-controlled unit and/or products contained therein, such asin the form of a thermographic image and/or in the form of transmitteridentifier and temperature value pairs.

In one implementation, upon reaching the latitude and longitudeprescribed by a waypoint adjacent a temperature-controlled unit, therobotic system can: orient itself according to a direction specified bythe waypoint (e.g., to face the temperature-controlled unit); and recorda set of static color images of the temperature-controlled unit in BlockS120, as shown in FIG. 3. Alternatively, in one implementation in whichthe autonomous vehicle navigates throughout the store without waypoints,the robotic system can: monitor its proximity to atemperature-controlled unit nearby by comparing its location to aplanogram of the store; align itself to the face of atemperature-controlled unit while navigating past thistemperature-controlled unit; and record a video or a sequence of frames(i.e., color images) of the temperature-controlled unit while navigatingpast this temperature-controlled unit.

If the temperature-controlled unit and/or adjacent waypoints are taggedwith a specification for thermal image collection, the robotic systemcan similarly record thermal images of the temperature-controlled unitvia the thermographic camera in Block S130, as described below.Additionally or alternatively, if the temperature-controlled unit and/oradjacent waypoints are tagged with a specification for querying nearbyRFID tags, the robotic system can record transmitter identifier andtemperature value pairs received from RFID tags located in thetemperature-controlled unit in Block S130, such after broadcasting aquery to these RFID tags as while navigating past thetemperature-controlled unit in Block S160 as described below.

Upon recording these data, the robotic system can navigate to a nextwaypoint in the sequence or to a next temperature-controlled unit andrepeat this process, as shown in FIG. 3.

The robotic system can then transmit these data to the remote system forremote processing, such as: in real-time when these data are recorded;in batches as the robotic system completes a subset of waypoints duringthe scan routine; or upon conclusion of the scan routine.

7. Product Detection

Block S140 of the method S100 recites identifying a set of productsstocked in the cooling unit based on features detected in the colorimage. Generally, in Block S140, the remote computer system (or therobotic system) can process color image data recorded by the roboticsystem during the scan routine to detect and identify products stockedthroughout the store, such as in both shelving structures, opentemperature-controlled units, and enclosed temperature-controlled units.For example, the remote computer system can implement methods andtechniques described in Ser. No. 15/600,527 to identify productscurrently stocked in the store based on features detected in these colorimages recorded by the robotic system.

In one example shown in FIGS. 1 and 3, upon receipt of a set of colorimages from the robotic system, the remote computer system can: detect afirst shelf in a first temperature-controlled unit in a first region ofa first color image—in the set of color images received—recorded by therobotic system at approximately a first time; identify an address of thefirst shelf; retrieve a first list of products assigned to the firstshelf by a planogram of the store based on the address of the firstshelf; retrieve a first set of template images from a database oftemplate images, wherein each template image in the first set oftemplate images depicts visual features of a product in the first listof products; extract a first set of features from the first region ofthe first color image; and determine that a unit of the first product isimproperly stocked on the first shelf in response to deviation betweenfeatures in the first set of features and features in the first templateimage. Then, in response to determining that the unit of the firstproduct is improperly stocked on the first shelf, the remote computersystem can generate a restocking prompt for the first product on thefirst shelf and serve this restocking prompt to an associate of thestore in real-time or append the restocking prompt to a currentrestocking list for the store, which the remote computer system laterserves the associate of the store upon conclusion of the inventorytracking routine, such as described in U.S. patent application Ser. No.15/600,527.

Furthermore, the remote computer system can; annotate a color image of atemperature-controlled unit with product identifiers (e.g., SKU values)of products thus detected in this color image; generate a spatial map ofthese product identifiers based on locations of corresponding productsdetected in the color image; and/or generate or update a realogram thatdepicts locations of the products. The remote computer system can thencompare spatial or georeferenced temperature data collected by therobotic system during this scan cycle to this annotated color image,spatial map, or realogram when mapping these temperature data toindividual products stocked in the temperature-controlled unit in BlockS142 described below.

The emote computer system can repeat the foregoing processes to derive acurrent stocking state and to generate annotated color images, spatialmaps, or realograms of other temperature-controlled units and/or otherinventory structures throughout the store based on other color imagesrecorded by the robotic system during this scan routine.

The remote computer system can also aggregate thesetemperature-controlled unit and inventory structures stocking states toquantify or qualify the total current stocking state of the store and togenerate prompts to correct or restock these temperature-controlledunits and/or inventory structures responsive to detecting deviation froma predefined planogram of the store. For example, the remote computersystem can: detect a first shelf of the cooling unit in a first regionof the color image; identify an address of the first shelf; based on theaddress of the first shelf, retrieve a first set of template images froma database of template images, wherein each template image in the firstset of template images depicts visual features of a product in the firstlist of products assigned to the first shelf by the planogram of thestore; extract a first set of features from the first region of thecolor image; and identify presence of a first subset of products, in thefirst list of products, on the first shelf based on alignment of thefirst set of features and template images in the first set of templateimages. However, the remote computer system can also: detect absence ofa first product, in the first list of products, on the first shelf inresponse to absence of features, in the first set of features, thatalign with template images, in the first set of template images,depicting the first product; generate a restocking prompt to restock thefirst shelf with units of the first product in response to detectingabsence of the first product on the first shelf; and serve thisrestocking prompt to a computing device affiliated with an associate ofthe store. Additionally or alternatively, the robotic system canaggregate deviation from the planogram into a global restocking list forthe store.

However, the remote computer system can implement any other method ortechnique to detect and identify individual products present intemperature-controlled units and other inventory structures depicted incolor images recorded by the robotic system during a scan routine.

8. Temperature Detection and Recordation

Blocks S140, S142, and S170 of the method S100 recite: identifying a setof products stocked in the cooling unit based on features detected inthe color image; mapping the set of temperatures to the set of productsat a first time during the first scan routine based on positions ofproducts in the set of products identified in the color image; andgenerating a record of temperatures of the set of products stocked inthe cooling unit at the first time, respectively. Generally, in BlocksS140, S142, and S170, the remote computer system (and/or the roboticsystem) can execute methods and techniques described below to calculateand store temperatures of individual products, slots, shelves, referencesurfaces, and/or the whole of temperature-controlled units throughoutthe store.

For example, upon detecting a unit of a product arranged in a slot on ashelf in a cooling unit and accessing or calculating a temperature ofthe product as described below, the remote computer system can append aproduct temperature database with a temperature of the product, a timeand date the temperature was measured, and a slot address for the unitof the product in Block S170. Similarly, the remote computer system canappend a slot temperature database with this temperature of the product,the time and date, and an address of this slot in thistemperature-controlled unit in Block S170. Furthermore, the remotecomputer system can append a shelf temperature database with average,minimum, maximum, and spatial variance temperatures of a set of producton the self, the time and date, and an address of this shelf in thistemperature-controlled unit. The remote computer system can also appenda temperature-controlled unit database with average, minimum, maximum,and spatial variance temperatures of products and/or reference surfacesin the temperature-controlled unit, the time and date, and an address orother identifier of this temperature-controlled unit.

The remote computer system can repeat this process over time based ondata recorded by the robotic system during subsequent scan routines inorder to generate timeseries of average, minimum, maximum, and/orspatial variance temperatures of products, slots, shelves, and/or thewhole of the temperature-controlled unit. From these timeseries data,the remote computer system can also estimate temporal variance ofaverage, minimum, or maximum temperatures of products, slots, shelves,and/or the whole of the temperature-controlled unit. From thesetimeseries data, the remote computer system can additionally oralternatively track total temperature exposure of individual productsstocked in the store over time, which may indicate a freshness orremaining shelf life of these individual products.

9. Open Temperature-Controlled Unit/Thermographic Camera

In one variation shown in FIG. 1, the robotic system: autonomouslynavigates toward an open cooling unit in Block S110; records a colorimage and a depth image of the open cooling unit in Block S120; recordsa thermal image of the open cooling unit in Block S130; and records anambient temperature, an ambient humidity, and an ambient light levelproximal the robotic system in Block S132. In particular, in thisvariation, the robotic system records a color image, a depth map, athermal image, and ambient data sufficient to enable the remote computersystem to estimate temperatures of products contained in an open coolingunit (or open heating unit) without access to data from additionaltemperature-enabled transmitters (e.g., temperature-enabled RFID tags)or other wireless infrastructure deployed to the open cooling unit.

In one implementation, upon arriving adjacent an open cooling unit(e.g., a door-less refrigeration unit) during a scan routine, therobotic system can record a thermal image of the cooling unit—via thethermographic camera—in Block S130 in addition to recording a colorimage of the cooling unit in Block S120. By aligning the thermal imageto products detected in the color image of the cooling unit—such asbased on a known offset between the color and thermographic cameras—theremote computer system can determine temperatures of these products atthe time these images were recorded.

However, ambient conditions, thermal emissivity or reflectivity of theseproducts or product packagings, and distances from the thermographiccamera in the robotic system to these products stored in thetemperature-controlled unit, etc. may introduce errors in interpretationof product temperatures directly from the thermal image. For example,products stocked in a cooling unit may not all exhibit the same thermalemissivity and may not radiate energy as perfect black bodies. A thermalimage of this cooling unit—recorded via the thermographic camera in therobotic system—may therefore contain thermal values that representcombinations of temperatures and thermal emissivities of these productsrather than temperatures of these products exclusively such that truetemperatures of these products differ from temperatures depicted in thethermal image by as much as several degrees Celsius.

The robotic system can therefore also record a depth image and ambientconditions while recording a color image and a thermal image of the opencooling unit in Blocks S120 and S130. The remote computer system canthen implement an atmospheric correction model (hereinafter a “physicsmodel”) to transform measures of thermal radiation from surfaces ofproducts stocked in the cooling unit—stored as pixel values in thethermal image—into temperatures of these products in Block S156 basedon: thermal radiation characteristics associated with these identifiedproducts; the ambient conditions; and distances from the thermographiccamera to these products calculated in Block S154, as shown in FIG. 1.

9.1 Thermal Coefficient

In particular, once these color, depth, thermal, and ambient data arerecorded by the robotic system and uploaded to the remote computersystem, the system can implement methods and techniques described aboveto identify products in these color images of temperature-controlledunits throughout the store. As described above, the remote computersystem can also: annotate these color images with product identifiers;generate spatial maps of products in these temperature-controlled units;or generate a realogram (or other virtual graphical representations) ofthe current stock states of these temperature-controlled units based onproducts identified in these color images. Then, once the remotecomputer system identifies a set of products in a particular coolingunit, the remote computer system can query a database of thermalcoefficients of products and product packagings for the thermalcoefficients (e.g., thermal emissivities) of each product identified inthis cooling unit. (Alternatively, product template images accessed bythe remote computer system to identify products depicted in the colorimage of the cooling unit can be annotated with known thermalcoefficients of products or product packagings depicted in this templateimages.)

Therefore, once the system identifies a product in a cooling unit, thesystem can leverage its derived knowledge of the identity of a productin a cooling unit to access a thermal coefficient of the product (e.g.,an emissivity of the product's packaging), which the system can laterpass into a physics model to transform or “correct” the temperature ofthe product read from the original thermal image.

9.2 Ambient Effects

Furthermore, a volume between a product in a cooling unit and therobotic system may be other than vacuum, which may introduce furthererror between the true temperatures of the product and a temperaturecontained in a region of the thermal image depicted this product. Theremote computer system can therefore also leverage other ambientdata—such as air temperature, air humidity, and ambient lightlevel—recorded by the robotic system concurrently with the thermal imageto further correct a temperature of the product read from the thermalimage. In particular, the system can fuse data from multiple sensorsintegrated into the robotic system to correct for assumptions (e.g.,blackbody radiation in a vacuum) of the thermographic camera to achievegreater accuracy and repeatability of measured temperatures of productsin cooling units throughout the store.

9.3 Mapping

In Block S142, the remote computer system can then map a set oftemperatures—extracted from the thermal image of the cooling unit andcorrected based on product emissivity, distance, ambient temperature,ambient humidity, and/or ambient light level, etc.—to correspondingproducts stocked in the store based on positions of these productsidentified in the color image of the open cooling unit.

In one implementation, the remote controller: selects a thermal image ofan open cooling unit recorded by the robotic system during a scanroutine; aligns the thermal image to a color image and to a depth imagerecorded substantially simultaneously by the robotic system, such asbased on known relative positions and orientations of the thermographiccamera, the color camera, and the depth sensor on the robotic system;implements methods and techniques described above to detect and identifyproducts depicted in the color image; delineates boundaries of eachproduct detected in the color image; tags each bounded region in thecolor image with product metadata, such as thermal coefficients (e.g.,thermal emissivities) associated with each of these products andaccessed from a product database or linked to template images of theseproducts, as described above; projects metadata and a boundary of eachproduct detected in the color image onto the thermal image to define aset of discrete regions in the thermal image that each depict oneproduct stocked in the cooling unit; and tags each discrete region inthe thermal image with ambient conditions—such as including airtemperature, air humidity, and ambient light level—recorded by therobotic system approximately concurrently with the thermal image.

For one discrete region depicting one product in the thermal image, theremote computer system can then: select a representative thermal pixel(e.g., a center pixel, an average of a center cluster of pixels) in thediscrete region of the thermal image; and calculate a pixel distancefrom the thermographic camera to the representative thermal pixel basedon a known distance between the thermographic camera and the depthsensor and based on a depth value stored in a depth pixel in the depthimage nearest the representative thermal pixel in the thermal image. Theremote computer system can then pass a thermal value (e.g., anuncorrected temperature) stored in the representative thermal pixel, thepixel distance, and the thermal coefficient and ambient conditionstagged to this region of the thermal image into a physics model. Thephysics model can thus output a corrected temperature of the productcoinciding with this discrete region of the thermal image based on theseinput values. Finally, the remote computer system can store this outputof the physics model as a corrected temperature of the product—depictedin this region of the thermal image—at a time that the thermal image wasrecorded by the robotic system.

In this implementation, the remote computer system can also repeat thisprocess to correct temperatures of multiple representative thermalpixels in one discrete region of the thermal image depicting one productin the open cooling unit. For example, the robotic system computersystem can: implement the foregoing process to transform thermal valuesstored in a set of ten pixels selected (randomly) within a region of thethermal image depicting one product; average or otherwise combine thesederived temperatures; and store this combined temperature as atemperature of the product at the time the thermal image was recorded.The remote computer system can then repeat this process for each otherdiscrete regions depicting a product facing in the thermal image.

In a similar implementation, rather than processing sets of singularthermal, color, and depth images, the remote computer system can compilemany thermal, color, and depth images recorded by the robotic systemduring a scan routine into composite thermal, color, and depth imagesrepresenting complete cooling units, complete rows of adjacent coolingunits, or complete aisles in the store. The remote computer system canthen implement methods and techniques described in the foregoingimplementation to: delineate regions of a composite thermal imagecorresponding to products detected in a composite color image (orindicated in a realogram or other spatial product map derived from acomposite color image); select representative thermal pixels in theseregions of the composite thermal image; and pass temperature values,distance data, thermal coefficients, and ambient data from the compositethermal and depth images into the physics model to calculate correctedtemperatures of products depicted in each of these product regions inthe composite thermal image.

Furthermore, the remote computer system can implement similar methodsand techniques to: detect reference surfaces in a cooling unit, such ascorners of the cooling unit or a shelf face in the cooling unit; and toderive corrected temperatures of these reference surfaces. The remotecomputer system can thus monitor corrected temperatures of thesereferences surfaces in the cooling unit over time in order to trackperformance of the cooling unit, predict maintenance, needs, and serveprompts for preemptive maintenance or replacement of the cooling unit tostore associates accordingly over time.

9.4 Mapping Example

Therefore, in one example, the robotic system can: autonomously navigatetoward an open cooling unit in Block S110; record a color image and adepth image of the open cooling unit in Block S120; record a thermalimage of the open cooling unit in Block S130; and record an ambienttemperature, an ambient humidity, and an ambient light level proximalthe robotic system in Block S132 during a scan routine. The roboticsystem can then upload these data to the remote computer system, such asin real-time or upon conclusion of the scan routine. Upon receipt ofthese data, the remote computer system can identify a stock keeping unit(or “SKU”) of each product in the set of products based on featuresdetected in the color image in Block S140, such as by matching thesefeatures in the color image to template images of products allocated tothis open cooling unit by the store planogram or based on a computervision model (or a machine learning, deep learning, or other perceptionmodel). The remote computer system can then: select a first productdepicted and identified in the color image; project a boundary of thefirst product onto the thermal image based on a known offset between thecolor camera and the thermographic camera on the robotic system; andproject the boundary of the first product onto the concurrent depth mapimage based on a known offset between the color camera and the depthsensor on the robotic system. The remote computer system can alsoretrieve a stored emissivity of a surface of the first product from adatabase based on an identity of the first product in Block S150, suchas by querying the database for the stored emissivity of packagingassociated with the first stock keeping unit of the first product. Theremote computer system can then: select a thermal pixel in the thermalimage depicting the first product; calculate a distance from the roboticsystem (or the thermographic camera in particular) to the first product(e.g., to a surface on the first product depicted by the thermal pixel)based on the depth image in Block S154; and then pass the emissivity ofthe surface of the first product, the thermal value stored in thethermal pixel, the distance, the measured ambient temperature proximalthe open cooling unit, the measured ambient humidity proximal the opencooling unit, and the measured ambient light level proximal the opencooling unit into a physics model to calculate the temperature of thefirst product at the first time in Block S156.

In this example, the remote computer system can implement a physicsmodel that includes the following equations to derive the correctedtemperature of the first product:

W_(tot) = ɛ ⋅ τ ⋅ W_(obj) + (1 − ϵ) ⋅ τ ⋅ W_(refl) + (1 − τ) ⋅ W_(alm) = ϵ ⋅ τ ⋅ σ ⋅ T_(obj)⁴ + (1 − ɛ) ⋅ τ ⋅ σ ⋅ T_(refl)⁴ + (1 − τ) ⋅ σ ⋅ T_(atm)⁴$T_{obj} = \sqrt[4]{\frac{W_{tol} - {( {1 - ɛ} ) \cdot \tau \cdot \sigma \cdot T_{refl}^{4}} - {( {1 - \tau} ) \cdot \sigma \cdot T_{atm}^{4}}}{ɛ \cdot \tau \cdot \sigma}}$${\tau( {d,\omega} )} = {{K_{atm} \cdot {\exp\lbrack {{- \sqrt{d}}( {\alpha_{1} + {\beta_{1}\sqrt{\omega}}} )} \rbrack}} + {( {1 - K_{atm}} ) \cdot {\exp( {{- \sqrt{d}}( {\alpha_{2} + {\beta_{2}\sqrt{\omega}}} )} \rbrack}}}$ω(ω_(%), K_(atm)) = ω_(%) ⋅ exp (h₁ + h₂ ⋅ T_(atm) + h₃ ⋅ T_(atm)² + h₄ ⋅ T_(atm)³)wherein: W_(tot) is the total radiation received by the thermographiccamera and depicted in the thermal image; ε·τ·W_(obj) is the emission ofthe product; (1−ε)·τ·W_(refl) is the emission of the surroundings andreflected by the product; (1−τ)·W_(atm) is the emission of theatmosphere (e.g., a function of ambient light level); σ is the StefanBoltzmann constant (5.67×10−8 W/m); c is the stored emissivity of thefirst product or first product packaging; τ is the transmittance of theatmosphere; T_(refl) is the reflected temperature; T_(atm) is theatmospheric/ambient temperature measured by the robotic system; d is thedistance from the thermal pixel to the thermographic camera; ω is acoefficient indicating content of water vapor in the atmosphere; ω % isthe relative humidity measured by the robotic system; K_(atm) is ascaling factor for the atmosphere damping; α₁, α₂ represent attenuationvalues for atmosphere without water vapor; β₁, β₂ represent attenuationvalues for water vapor; and T_(obj) is the temperature of the product.

The remote computer system can: execute this process to derive acorrected temperature of each other product detected in the color image;and then store these corrected temperatures, such as in a database oftimestamped temperature values for individual products, slots, orshelves in the open cooling unit or for the open cooling unit generally,such as described below. For example, the remote computer system canaggregate corrected temperature derived from data recorded by therobotic system during multiple scan routines executed over time (e.g.,days, weeks, months) into timeseries of temperatures in each slot in theopen cooling unit over this period of time.

10. Short-Range Wireless Communication Tags

In one variation, enclosed cooling units throughout the store areoutfitted with temperature-enabled transmitters, such as RFID (or NFC)tags with integrated temperature sensors, each configured to broadcast aunique electronic identifier and a temperature value read from itsintegrated temperature sensor upon receipt of an excitation signal(e.g., from a wireless transmitter integrated into the robotic system).In particular, a glass door on a cooling unit may optically obscuresurfaces of products stocked inside the cooling unit from athermographic camera integrated into the robotic system. Morespecifically, the glass in the door of a cooling unit may be opaque tothe thermal radiation emanating from products stored inside the coolingunit, thereby inhibiting temperature monitoring of these products viathe thermographic camera.

Therefore, one or more temperature-enabled RFID (or similar) tags can bearranged inside enclosed cooling units (e.g., freezers and refrigeratorswith doors) throughout the store, and the robotic system can readtemperatures from these temperature-enabled RFID tags. For example, atemperature-enabled RFID tag can be arranged at or adjacent each slot inan enclosed cooling unit. In another example, a linear array oftemperature-enabled RFID tags is arranged across each shelf in thecooling unit, such as one RFID tag per linear meter of horizontal shelfper cooling unit. In a similar example, a grid array oftemperature-enabled RFID tags is arranged throughout a cooling unit,such as according to a horizontal offset of approximately one meter anda vertical offset of approximately one-half meter between adjacenttemperature-enabled RFID tags.

Furthermore, in this variation, the robotic system can include one ormore readers (e.g., wireless tag interrogator, RFID readers) configuredto broadcast an excitation signal and to subsequently receive a uniqueelectronic identifier paired with temperature values from nearbytemperature-enabled transmitters.

However, open and enclosed cooling units throughout the store can beoutfitted (e.g., retrofitted) with temperature-enabled transmitters ofany other type and in any other arrangement, and the robotic system canbe configured to retrieve temperature values from thesetemperature-enabled transmitters in any other way.

10.1 Scanning

Therefore, in this variation, the robotic system can: autonomouslynavigate toward the cooling unit comprising an enclosed cooling unit inBlock S110; record a color image of the enclosed cooling unit (since aglass door of the enclosed cooling unit may still be substantiallytransparent to the color camera in the robotic system) in Block S120;broadcast a query to a set of temperature-enabled transmitters arrangedin the enclosed cooling unit in Block S160; and record a set oftemperature values received from the set of temperature-enabledtransmitters in Block S130 during a scan routine, as shown in FIGS. 1and 3.

In one implementation, when occupying a waypoint specifying collectionof temperature data from nearby RFID tags during a scan routine, therobotic system can: broadcast a query for temperature values from a setof temperature-enabled transmitters arranged in thetemperature-controlled unit in Block S160; and record a set oftemperature values transmitted by the set of temperature-enabledtransmitters in Block S130. In particular, the robotic system can:sequentially navigate through a sequence of waypoints defined for thestore, as described above, in Block S110. For each waypoint specifyingcollection of temperature data from nearby RFID tags, the robotic systemcan also broadcast an excitation signal via the RFID reader in BlockS160 and thus collect unique electronic identifiers and temperaturevalues from RFID tags nearby in Block S130, as shown in FIG. 1.Therefore, in this implementation, the robotic system can: interrogate aset of temperature-enabled RFID tags arranged in the enclosed coolingunit in Block S160 in response to occupying a particular waypoint thatindicates proximity of temperature-enabled RFID tags or in response tooccupying a location in the store near a temperature-controlled unitknown to contain temperature-enabled RFID tags; and then record uniqueelectronic identifier and temperature value pairs received from theseRFID tags in Block S130.

(Alternatively, self-powered (e.g., battery-powered, wall-powered)temperature-enabled tags can be arranged in enclosed cooling unitsthroughout the store, and the robotic system can broadcast query signalsto these self-powered temperature-enabled tags and recorded temperaturevalues thus returned by these self-powered temperature-enabled tags.)

10.2 RFID Tag Location

The robotic system can also derive locations of these RFID tags (orother temperature-enabled transmitters) in the enclosed cooling unit andstore temperature values received during this scan routine with theircorresponding tag locations prior to uploading these temperature data tothe remote computer system.

In one implementation, the robotic system: calculates an origin of asignal—carrying a unique electronic identifier and temperature valuebroadcast by an RFID tag in a cooling unit—relative to the roboticsystem (e.g., relative to an antenna of the RFID reader integrated intothe robotic system). For example, the robotic system can implementtriangulation techniques or phase-of-arrival techniques to calculate theorigin—in three-dimensional space—of such a signal relative to therobotic system. The robotic system can then calculate an absolutegeospatial location (e.g., latitude, longitude, and height above thefloor of the store) within a coordinate system assigned to the storebased on the calculated origin of the signal relative to the roboticsystem and the position and orientation of the robotic system in thiscoordinate system at a time that the signal was received by the roboticsystem. The robotic system can then tag the unique electronic identifierand temperature value carried by this signal with this absolutegeospatial location in the store.

In another implementation, temperature-enabled RFID tags are labeledwith unique optical fiducials, such as unique barcodes or quick-response(or “QR”) codes. A lookup table or other database can also define linksbetween unique optical fiducials and unique electronic identifiersbroadcast—with temperature values—by these RFID tags when queried (or“interrogated”). In this implementation, a temperature-enabled RFID tagcan be installed (e.g., adhered) onto a temperature-controlled unit withits unique optical fiducial facing outwardly toward an adjacent aislesuch that the optical fiducial falls into the field of view of a colorcamera on the robotic system when the robotic system faces thistemperature-controlled unit. Therefore, when the robotic system capturesa color image of the temperature-controlled unit during a scan routinein Block S120, this color image may depict the unique optical fiducialson this RFID tag. Furthermore, upon receipt of this color image and aset of transmitter identifier and temperature value pairs recorded bythe robotic system during this scan routine, the remote computer systemcan: detect a set of optical fiducials representing these RFID tags inthe color image; query the lookup table or other database with theseoptical fiducials in order to retrieve unique electronic identifierslinked to each of these optical fiducials; and link temperature valuesreceived from these RFID tags during the scan routine to locations—onthe temperature-controlled unit—of corresponding optical fiducialsdetected in the color image. In particular, for each temperature valuein the set of temperature values received from the robotic system, theremote computer system can link the temperature value to a location ofan optical fiducial—in the set of optical fiducials detected in thecolor image—associated with the unique electronic identifier receivedwith this temperature value. Therefore, in this implementation, theremote computer system can detect and identify RFID tags directly in acolor image of a temperature-controlled unit and then link temperaturevalues received from these RFID tags to discrete 3D locations in thestore (and in the temperature-controlled unit more specifically) basedon known associations between optical fiducials on these RFID tags andelectronic identifiers broadcast by these RFID tags.

Alternatively, absolute geospatial locations—within the coordinatesystem assigned to the store—of each temperature-enabled RFID tag can bestored in a lookup table or other database. Upon receipt of a uniqueelectronic identifier and a temperature value from a temperature-enabledRFID tag, the robotic system can: query the lookup table for a knownlocation (e.g., latitude, longitude, and height in the coordinate systemof the store) of a tagged assigned this unique electronic identifier;and then tag a temperature value—received with this unique electronicidentifier—with this location. Alternatively, the robotic system canupload unique electronic identifier and temperature value pairs—recordedduring the scan routine—to the remote computer system, and the remotecomputer system can associate these temperature values with physicallocations in the store (and the enclosed cooling unit more specifically)based on stored locations of these unique electronic identifiers.

In this implementation, locations of RFID tags can be entered manually,such as into a planogram of the store by a store associate or by anoperator affiliated with the robotic system. Alternatively, during scanroutines executed by the robotic system over time, the robotic systemcan: collect unique electronic identifier and temperature value pairsfrom these RFID tags; derive locations of these RFID tags as describedabove; and combine RFID tag location derived over multiple scan routinesinto a final verified location of these RFID tags. The robotic system(or the remote computer system) can then store these final RFID taglocations in a lookup table and transition to linking temperature valuesreceived from these RFID tags to physical locations in the store basedon RFID tag locations stored in this lookup table.

10.3 Temperature Mapping

In Block S142, the remote computer system can then map the set oftemperatures received from these RFID tags to the set ofproducts—detected in the color image in Block S140 as describedabove—based on: locations of the set of RFID tags in the enclosedcooling unit; and locations of the set of products identified in thecolor image. In particular, the robotic system and/or the remotecomputer system can link temperature values received from RFID tagsthroughout the store by the robotic system during the scan routine: tophysical locations throughout the store; or to positions withintemperature-controlled units at known locations through the store morespecifically. The remote computer system can then compile thesegeoreferenced temperature data to generate a map or other representationof temperatures throughout these temperature-controlled units and thusderive more specific temperatures of shelves, slots, or individualproducts stocked in these temperature-controlled units.

In one implementation, the remote computer system: generates a realogramor other spatial map of products identified in the color image of thetemperature-controlled unit; projects georeferenced temperaturesreceived from RFID tags in the temperature-controlled unit onto thisrealogram or other spatial map of products; associates each product withthe nearest georeferenced temperature in Block S142; and recordstemperature value and product identifier (e.g., SKU) pairs for theseproducts and with a time of the scan routine in Block S170. For example,the remote computer system can: calculate a temperature gradient acrossthe enclosed cooling unit by interpolating between the set oftemperature values based on known or detected locations of correspondingRFID tags in the enclosed cooling unit; project boundaries of the set ofproducts—identified in the color image—onto the temperature gradient;and, for each product in this set of products, extract a temperature ofthe product from a region of the temperature gradient contained withinthe boundary of the product thus projected onto the temperaturegradient. Therefore, in this example, the remote computer system can mapa temperature value received from a temperature-enabled RFID to a subsetof products—in a set of products detected in the color image of thetemperature-controlled unit—proximal the known or detected location ofthe temperature-enabled RFID tag.

Similarly, the remote computer system can: project georeferencedtemperatures received from RFID tags in the temperature-controlled unitonto a planogram or slot map of the temperature-controlled unit;associate each slot in the temperature-controlled unit with the nearestgeoreferenced temperature (e.g., the temperature received an RFID taglocated in a nearest or store location in the temperature-controlledunit) in Block S142; and record temperature value and slot address pairsfor the temperature-controlled unit and with the time of the scanroutine in Block S170. The remote computer system can additionally oralternatively record temperature value and shelf address pairs for thetemperature-controlled unit and with the time of the scan routine inBlock S170.

Alternatively, the robotic system can interpolate between temperaturesreceived from RFID tags in the temperature-controlled unit and/orextrapolate temperatures beyond these RFID tags and estimatetemperatures of products, slots, and/or shelves in thetemperature-controlled unit accordingly. In one implementation, theremote computer system: receives a color image of an enclosed coolingunit and electronic identifiers and temperature value pairs recorded bythe robotic system at a first time during a scan routine; links thesetemperature values to discrete locations within the enclosed coolingunit in the store based on corresponding electronic identifiers, asdescribed above; and calculates a temperature gradient across theenclosed cooling unit at the first time by interpolating between thesetemperature values based on known or detected locations of theircorresponding RFID tags, as shown in FIG. 3. In this implementation, theremote computer system also: detects products in the color image inBlock S140; generates a realogram or other spatial map of these productsin the enclosed cooling unit; projects the temperature gradient onto therealogram or other spatial product map (e.g., based on known or detectedlocations of the RFID tags in the enclosed cooling unit) (or viceversa); extracts an average temperature, peak temperature, minimumtemperature, and/or spatial temperature variance, etc. of each productdepicted in this realogram or other spatial product map; and recordsthese temperature value and product identifier pairs with the time ofthe scan routine in Block S170. In this implementation, the remotecomputer system can additionally or alternatively: project thetemperature gradient onto a slot map of the enclosed cooling unit (e.g.,based on known or detected locations of the RFID tags in the enclosedcooling unit) (or vice versa); extract an average temperature, peaktemperature, minimum temperature, and/or spatial temperature variance,etc. of each slot represented in the slot map; and record thesetemperature value and slot address pairs with the time of the scanroutine in Block S170. The remote computer system can additionally oralternatively: calculate average temperature, peak temperature, minimumtemperature, and/or spatial temperature variance, etc. of each shelf inthe enclosed cooling unit; and record temperature value and shelfaddress pairs for the enclosed cooling unit with the time of the scanroutine in Block S170. Similarly, the remote computer system can:calculate an average temperature, peak temperature, minimum temperature,and/or spatial temperature variance, etc. of the enclosed cooling unitas a whole or of discrete reference surfaces at known locations withinthe enclosed cooling unit; and record these temperature values with anidentifier or address of the enclosed cooling unit and the time of thescan routine in Block S170.

10.4 Store-wide Temperature Map

In another implementation, the remote computer system: populates a 3Dtemperature map of the store with points representing locations of eachtemperature-enabled RFID tag in the store; labels each point with thetemperature value received from the corresponding temperature-enabledRFID tag by the robotic system during the last scan routine; insertsboundaries of discrete cooling units into the temperature map, such asby projecting a floor plan of the store—including perimeters of thesecooling units—or a planogram of the store onto a ground plane in thetemperature map; and then defines discrete groups of points containedwithin the bounds of each cooling unit or within the bounds of a row ofadjacent cooling units (e.g., a row of open or closed cooling unitsspanning one side of one aisle) in the store. For a group of points thusbounded by a single cooling unit, the remote computer system caninterpolate temperatures in the cooling unit between points in thisgroup and extrapolate temperatures from these points to the 2D or 3Dboundary of the cooling unit. Therefore, in this example, the remotecomputer system can calculate a temperature gradient or temperaturelines extending across a vertical plane spanning an opening (e.g., adoor, a window) of the cooling unit (or row of cooling units).

The remote computer system can then overlay this temperature gradientonto a realogram of the cooling unit (or row of adjacent cooling units)to determine temperatures of products detected in this cooling unit. Theremote computer system can repeat this process for each other group ofpoints corresponding to other cooling units throughout the store todetermine temperatures of products detected in these other coolingunits.

Alternatively, once the remote computer system has insertedtemperature-labeled points into the temperature map of the store, theremote computer system can: access a planogram of the store; project aset of slot points—representing locations of slots in cooling unitsthroughout the store—into the temperature map; and define discretegroups of temperature-labeled points corresponding to discrete coolingunits (or connected rows of adjacent cooling units) throughout thestore, as described above. For each cooling unit (or row of adjacentcooling units), the remote computer system can then interpolate orextrapolate a temperature at each slot point—representing a slot in thecooling unit—according to values of temperature-labeled points in thecorresponding group, thereby generating a map of temperatures at eachproduct slot (e.g., at the vertical and horizontal center of a foremostface of the product slot) in the cooling unit. The remote computersystem can repeat this process for each other cooling unit or row ofadjacent cooling units throughout the store to generate maps oftemperatures at product slots in these other cooling units.

In a similar implementation, once the system transforms color images ofcooling units recorded by the robotic system during the scan routineinto realograms of types and positions of products on these coolingunits throughout the store, as described above, the system can: projecttemperature-labeled points into these realograms according to knownlocations of corresponding temperature-enabled RFID tags and knownlocations of cooling units in the store; calculate centers of eachproduct facing represented in these cooling unit realograms; and theninterpolate or extrapolate a temperature at each product center fromthese temperature-labeled points, as described above.

However, the system can implement any other method or technique totransform discrete temperature values received from temperature-enabledRFID tags coupled to cooling units throughout the store into atemperature map of the store, into temperatures of discrete productslots in these cooling units, or into temperatures of discrete productsstocked in these cooling units.

11. Temperature Errors

In one variation shown in FIG. 1, the remote computer system also checksthat product and/or temperature-controlled unit temperatures meetpredefined temperature requirements for these products or cooling units.

11.1 Absolute Cooling Unit Temperature Errors

In one implementation, for each cooling unit, the remote computer systemretrieves a target temperature or target temperature range for thetemperature-controlled unit from a database, such as from a lookuptable, and selectively prompts inspection of the temperature-controlledunit responsive to a measured temperature of the temperature-controlledunit deviating from this target temperature or target temperature range.In one example, the remote computer system can calculate a meantemperature of a temperature-controlled unit during the last scanroutine, such as by: averaging a set of temperatures received from RFIDtags in the temperature-controlled unit; averaging temperatures of eachproduct in the temperature-controlled unit derived from a thermal image;or averaging or combining temperatures across a temperature gradientcalculated by the remote computer system. The remote computer system canalso access a current temperature setting of the cooling unit, such asby: reading a temperature from a temperature setting readout on thetemperature-controlled unit depicted in the concurrent color image ofthe temperature-controlled unit; or reading a temperature settingassigned to the cooling unit by the planogram of the store. Then, inresponse to this mean temperature of the temperature-controlled unitdeviating from the current temperature setting by more than a thresholddifference (e.g., 2° C.), the remote controller can serve a maintenanceprompt to inspect the temperature-controlled unit to a computing deviceaffiliated with an associate of the store. For example, the remotecomputer system can: serve this prompt to a store associates smartphoneor tablet in real-time upon detecting this temperature deviation (e.g.,if this temperature deviation is large); or append a global restockinglist for the store with this prompt (e.g., if this temperature deviationis small) and later serve this global restocking list for the store tostore associates during a schedule restocking period (e.g., between 10PM and 12 AM daily).

Alternatively, in response to this mean temperature of thetemperature-controlled unit deviating from the current temperaturesetting by more than a threshold difference (e.g., 2° C.), the remotecontroller can: trigger the remote computer system to rescan thetemperature-controlled unit, such as one hour later; repeat theforegoing processes to derive a second mean temperature of thetemperature-controlled unit; and then serve this prompt to the storeassociate to inspect the temperature-controlled unit if the second meantemperature of the temperature-controlled unit still deviates from thecurrent temperature setting by more than the threshold difference.

In a similar example, the remote computer system can derive arepresentative temperature of a temperature-controlled unit, such as: amean temperature within a temperature gradient of a vertical plane ofthe temperature-controlled unit; a temperature of a reference surface inthe temperature-controlled unit; or an average temperature of alldetected products in the temperature-controlled unit. The remotecomputer system can then issue a prompt or notification to inspect atemperature-controlled unit and either adjust a temperature setting onthe temperature-controlled unit or schedule maintenance of therepresentative temperature of the temperature-controlled unit exceedsthe target temperature or a target temperature range assigned totemperature-controlled unit by a threshold maintenance difference (e.g.,2° F.). However, if the representative temperature of thetemperature-controlled unit exceeds the target temperature or targettemperature range assigned to the temperature-controlled unit by athreshold failure difference (e.g., 5° F.), the remote computer systemcan serve a prompt to a store associate (e.g., a store manager) toimmediately transfer product out of the temperature-controlled unit andinto another temperature-controlled unit before these products spoil orare otherwise overexposed to heat overage (e.g., for refrigerated orfrozen goods in a cooling unit) or heat scarcity (e.g., for hot goods ina heated display).

Furthermore, if only one product or a small proportion of products in anon-linear (e.g., “random”) arrangement of slots in the cooling unit aredetected at temperatures above the target temperature range of thecooling unit, the remote computer system can determine that theseproducts are exhibiting elevated temperatures: as a result of beingrecently stocked in the cooling unit; or as a result of being removedfrom the cooling unit and then recently being returned to the coolingunit. The remote computer system can therefore avoid flagging thecooling unit as malfunctioning when only a small proportion of productsdetected in the cooling unit exhibit elevated temperatures.

However, in this implementation, the remote computer system canautomatically submit a request for maintenance of atemperature-controlled unit if a temperature-controlled unit deviatesfrom assigned target temperature or target temperature range in anyother way.

10.2 Cooling Unit Temperature Variance Errors

In another implementation, the remote computer system selectivelyprompts store associates to inspect, repair, or replace atemperature-controlled unit based on a spatial and/or temporaltemperature variance.

In one example, the remote computer system: implements methods andtechniques described above to calculate a temperature gradient across avertical plane of a temperature-controlled unit based on a thermal imageof the temperature-controlled unit or based on temperature valuesreceived from temperature-enabled wireless transmitters arranged in thetemperature-controlled unit. The remote computer system then calculatesa spatial temperature variance across this plane of thetemperature-controlled unit based on the temperature gradient, such asby: filtering the temperature gradient to reduce noise; extractingminimum and maximum temperatures from the filtered temperature gradient;and calculating a temperature difference between the minimum and maximumtemperatures. Because a spatial temperature variance may indicatefailure of a fan, a door ajar, or other failure in atemperature-controlled unit, the remote computer system can serve amaintenance prompt to inspect the cooling unit to a computing deviceaffiliated with an associate of the store if the spatial temperaturevariance of the temperature-controlled unit exceeds a threshold variance(e.g., 3° C.).

In this example, the remote computer system can prompt a store associateto check a cooling unit with a row of doors for a door ajar condition ifthe spatial temperature variance occurs laterally across the temperaturegradient of the cooling unit. Alternatively, the remote computer systemcan prompt a store associate to inspect a fan in the cooling unit forfailure or intermittent operation if the spatial temperature varianceoccurs vertically in the temperature gradient of the cooling unit.

Alternatively, in the foregoing example, in response to the temperaturevariance of a temperature-controlled unit exceeding a thresholdvariance, the remote controller can: trigger the remote computer systemto rescan the temperature-controlled unit, such as one hour later;repeat the foregoing processes to derive a second temperature varianceof the temperature-controlled unit; and then serve a prompt to the storeassociate to inspect the temperature-controlled unit if the secondtemperature variance of the temperature-controlled unit still exceedsthe threshold variance.

10.3 Absolute Product Temperature Error

In a similar implementation, upon detecting a unit of a product in animage of a cooling unit, the system can retrieve a target temperature ortarget temperature range assigned to the product. If the received orcalculated temperature of the unit of the product exceeds the targettemperature or target temperature range, the system can flag a slot inthe cooling unit containing the unit of the product and selectivelyserve notifications related to the slot and/or to the unit of theproduct specifically to associates of the store. For example, if thetemperature of the product exceeds a maximum threshold temperature(e.g., 75° F. for dairy products) at any time, the system can serve anotification—to associates of the store—including: an address of theslot; a prompt to remove the product from the slot and to discard theproduct; and a prompt to check products behind and around this productfor temperature variations (e.g., with a handheld infrared thermometer).

In a similar implementation, the remote computer system can: identify aproduct in a color image of a temperature-controlled unit recorded bythe robotic system during a scan routine; extract or derive atemperature of the product based on temperature-related data collectedby the robotic system during this scan routine; and store thistemperature of the product, such as with the time of this scan routineand a slot address occupied by the product, in a store record. Theremote computer system can also: access a temperature requirement of theproduct; verify that the temperature of the product—during this scanroutine—falls within a temperature range specified for the product inthe temperature requirement; and flag the product in the store record ifthe temperature of the product falls outside of this temperature range.Furthermore, if the temperature of the product falls outside of thetemperature range, the robotic system can: initialize an electronicnotification; identify the product in the electronic notification;identify the temperature-controlled unit and a slot address of theproduct in the electronic notification; write a prompt to remove (anddiscard) the product from the temperature-controlled unit to theelectronic notification; and then serve the electronic notification toan associate of the store, such as to the associate's smartphone ortablet in real-time. Alternatively, the robotic system can append aprompt—to remove the product from the temperature-controlled unit—to anext global restocking list for the store.

10.4 Accumulated Product Temperature Error

In one variation in which the robotic system is regularly scanningtemperature-controlled units and cooling units throughout the store, thesystem can implement the foregoing methods and techniques to derivetemperatures of cooling units and products stocked in these coolingunits during each scan routine. In this variation, the system can flag aparticular product in a cooling unit if the temperature of thisproduct—derived from data recorded during a first scan routine—exceeds atarget temperature or target temperature range assigned to this product.If the temperature of this product—derived from data recorded during asubsequent scan routine (e.g., thirty minutes later)—is less than thetarget temperature or falls within the target temperature range, thesystem can clear this flag for the product. However, if the temperatureof the product—derived from data recorded during the second scanroutine—still exceeds the target temperature or target temperaturerange, the system can immediately serve a prompt to an associate of thestore to remove the product from the cooling unit due to possibleover-temperature exposure or serve a prompt to an associate of the storeto inspect the cooling unit (e.g., if products in several adjacent slotssimilarly exhibit over-temperatures), as described above.

Alternatively, as the robotic system executes additional scan routinesand the system derives additional temperatures of the product from datarecorded by the robotic system during these scan routines, the systemcan integrate temperatures of the product—exceeding target temperaturesor temperature ranges assigned to the product—over time and thusestimate temperature exposure (or “over-temperature”) of the product (orproducts in the corresponding slot more generally). When the temperatureexposure of the product (or products in the slot more generally) exceedsa threshold exposure, the system can then serve a prompt to an associateof the store to remove and replace the product, remove and replace allproducts in the corresponding slot, and/or inspect the cooling unit,etc.

In a similar implementation, the remote computer system can: repeat theforegoing processes to detect or derive temperatures of products in atemperature-controlled unit from temperature-related data recorded bythe robotic system over a series of scan routines within the store;aggregate timestamped temperatures of these products; and estimatetemperature exposures of these products by integrating these timestampedtemperatures over time. For example, the remote computer system canestimate temperature exposures of products—in a set of products detectedin color images of a temperature-controlled unit—between a first timeand a second time by integrating between: a first set of temperatures ofthe set of products derived from temperature-related data recordedduring a first scan routine at a first time; and a second set oftemperatures of the set of products derived from temperature-relateddata recorded during a second scan routine at a second time; based on atime duration between the first time and the second time.

In the foregoing implementation, the remote computer system can also:access a shelf life model of a particular product in this set ofproducts detected in a last color image of a temperature-controlledunit; estimate a temperature exposure of the particular product—such asin the form of a degree-Celsius-hour value—as described above; andestimate a remaining shelf life of the particular product based on theshelf life model and the temperature exposure of the particular product.For example, the remote computer system can access a shelf life modelthat specifies a total permitted temperature exposure of a product, suchas in the form of: a fixed degree-Celsius-hour value (e.g., 140degree-Celsius-hours for raw meats); or a parametric model. In thisexample, a parametric shelf life model can define: a long total storagetime for a product if frozen (e.g., weeks for a meat product); amedium-duration time trigger for product sale once its temperatureexceeds a low temperature threshold (e.g., two days if the temperatureexceeds 1° C. for the meat product); a short-duration time trigger forproduct sale once its temperature exceeds a medium temperature threshold(e.g., four hours if the temperature exceeds 4° C. for the meatproduct); and a discard trigger for the product once its temperatureexceeds a high temperature threshold (e.g., if the temperature exceeds10° C.).

The remote computer system can thus compare the temperature exposure ofthe particular product to the shelf life model to estimate a remainingshelf life of the particular product, such as in the form of remainingpermitted degree-Celsius-hour exposure. For example, the remote computersystem can calculate a remaining permitted degree-Celsius-hour exposurefor the particular product and then multiply this value by a lastdetected or derived temperature of the product in order to estimate aremaining shelf life (e.g., in hours) of the particular product. Then,if this estimated remaining shelf life of the particular product fallsbelow a threshold duration (e.g., four hours for a meat product; one dayfor a dairy product), the remote computer system can: generate anelectronic notification identifying the particular product, the addressof the slot occupied by the particular product in thetemperature-controlled unit, and the remaining shelf life (e.g., inhours) of the particular product; and then serve the electronicnotification to a computing device affiliated with an associate of thestore, such as in real-time. Alternatively, the remote computer systemcan note this remaining shelf life of the particular product in theglobal restocking list for the store. The remote computer system cansimilarly serve a prompt to the store associate to discard theparticular product—such as in real-time—if the remaining shelf life ofthe particular product thus calculated is null.

However, the remote computer system can detect and handle temperaturedeviations of temperature-controlled units, product slots, or specificproducts in these temperature-controlled units in any other way andaccording to any other schema.

The systems and methods described herein can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated with the application, applet, host, server, network, website,communication service, communication interface,hardware/firmware/software elements of a user computer or mobile device,wristband, smartphone, or any suitable combination thereof. Othersystems and methods of the embodiment can be embodied and/or implementedat least in part as a machine configured to receive a computer-readablemedium storing computer-readable instructions. The instructions can beexecuted by computer-executable components integrated bycomputer-executable components integrated with apparatuses and networksof the type described above. The computer-readable medium can be storedon any suitable computer readable media such as RAMs, ROMs, flashmemory, EEPROMs, optical devices (CD or DVD), hard drives, floppydrives, or any suitable device. The computer-executable component can bea processor but any suitable dedicated hardware device can(alternatively or additionally) execute the instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

We claim:
 1. A method for monitoring cooling units in a storecomprising: at a robotic system, during a first scan routine:autonomously navigating toward a cooling unit in the store; recording acolor image of the cooling unit; and scanning a set of temperatureswithin the cooling unit; identifying a set of products stocked in thecooling unit based on features detected in the color image; mapping theset of temperatures to the set of products at a first time during thefirst scan routine based on positions of products in the set of productsidentified in the color image; and generating a record of temperaturesof the set of products stocked in the cooling unit at the first time. 2.The method of claim 1: wherein autonomously navigating toward thecooling unit comprises autonomously navigating toward the cooling unitcomprising an open cooling unit; wherein scanning the set oftemperatures within the cooling unit comprises, during the first scanroutine, recording a thermal image of the open cooling unit; and whereinmapping the set of temperatures to the set of products comprises, for afirst product in the set of products identified in the color image:retrieving a stored emissivity of a surface of the first product from adatabase based on an identity of the first product; selecting a thermalpixel in the thermal image depicting the first product; and calculatinga temperature of the first product at the first time based on a thermalvalue stored in the thermal pixel and the emissivity of the surface ofthe first product.
 3. The method of claim 2: further comprising, duringthe first scan routine: recording a depth image of the open coolingunit; and recording an ambient temperature, an ambient humidity, and anambient light level proximal the robotic system; further comprisingcalculating a distance from the robotic system to the first productbased on the depth image; and wherein calculating the temperature of thefirst product at the first time comprises passing the emissivity of thesurface of the first product, the thermal value stored in the thermalpixel, the distance, the ambient temperature, the ambient humidity, andthe ambient light level into a physics model to calculate thetemperature of the first product at the first time.
 4. The method ofclaim 2: wherein identifying the set of products stocked in the coolingunit comprises identifying a stock keeping unit of each product in theset of products based on features detected in the color image; andwherein retrieving the stored emissivity of the surface of the firstproduct comprises querying the database for the stored emissivity ofpackaging associated with a first stock keeping unit of the firstproduct.
 5. The method of claim 1: wherein autonomously navigatingtoward the cooling unit comprises autonomously navigating toward thecooling unit comprising an enclosed cooling unit; wherein scanning theset of temperatures within the cooling unit comprises, during the firstscan routine: broadcasting a query to a set of temperature-enabledtransmitters arranged in the enclosed cooling unit; and recording a setof temperature values received from the set of temperature-enabledtransmitters; and wherein mapping the set of temperatures to the set ofproducts comprises: accessing locations of the set oftemperature-enabled transmitters in the enclosed cooling unit; andmapping the set of temperature values received from the set oftemperature-enabled transmitters to the set of products at the firsttime based on: locations of the set of temperature-enabled transmittersin the enclosed cooling unit; and locations of the set of productsidentified in the color image.
 6. The method of claim 5: whereinbroadcasting the query for temperature values from the set oftemperature-enabled transmitters comprises interrogating the set oftemperature-enabled transmitters comprising a set of temperature-enabledradio frequency identification tags arranged in the enclosed coolingunit; and wherein mapping the set of temperature values received fromthe set of temperature-enabled transmitters to the set of products atthe first time comprises: calculating a temperature gradient across theenclosed cooling unit by interpolating between the set of temperaturevalues based on locations of corresponding temperature-enabledtransmitters, in the set of temperature-enabled transmitters, in theenclosed cooling unit; projecting boundaries of the set of products,identified in the color image, onto the temperature gradient; and foreach product in the set of products, extracting a temperature of theproduct from a region of the temperature gradient contained within theboundary of the product projected onto the temperature gradient.
 7. Themethod of claim 5: wherein recording the set of temperature valuesreceived from the set of temperature-enabled transmitters comprisesrecording the set of temperature values received from the set oftemperature-enabled transmitters, each temperature value in the set oftemperature values paired with an identifier; and wherein accessinglocations of the set of temperature-enabled transmitters in the enclosedcooling unit comprises: detecting a set of optical fiducialsrepresenting the set of temperature-enabled transmitters in the colorimage; and for each temperature value in the set of temperature values,linking the temperature value to a location of an optical fiducial, inthe set of optical fiducials, associated with the identifier receivedwith the temperature value.
 8. The method of claim 5: whereinautonomously navigating toward the cooling unit in the store comprises,during the first scan routine, autonomously navigating through asequence of waypoints defined within the store, the sequence ofwaypoints comprising a particular waypoint adjacent the cooling unit;and wherein scanning the set of temperatures within the cooling unitcomprises broadcasting the query for temperature values in response tooccupying the particular waypoint that indicates proximity oftemperature-enabled transmitters.
 9. The method of claim 5, furthercomprising: calculating a temperature gradient across the enclosedcooling unit by interpolating between the set of temperature valuesbased on locations of corresponding temperature-enabled transmitters, inthe set of temperature-enabled transmitters, in the enclosed coolingunit; calculating a spatial temperature variance in the temperaturegradient across the enclosed cooling unit at the first time; and inresponse to the spatial temperature variance of the cooling unitexceeding a threshold variance, serving a maintenance prompt to inspectthe cooling unit to a computing device affiliated with an associate ofthe store.
 10. The method of claim 1: wherein autonomously navigatingtoward the cooling unit comprises autonomously navigating toward thecooling unit comprising an enclosed cooling unit; wherein scanning theset of temperatures within the cooling unit comprises, during the firstscan routine: broadcasting a query to a temperature-enabled transmittercoupled to the enclosed cooling unit; and recording a temperature valuereceived from the temperature-enabled transmitter; and wherein mappingthe set of temperatures to the set of products comprises mapping thetemperature value received from the temperature-enabled transmitter to asubset of products, in the set of products, proximal a known location ofthe temperature-enabled transmitter.
 11. The method of claim 1, furthercomprising: accessing a temperature requirement of a first product inthe set of products detected in the color image; based on the record,verifying that a temperature of the first product at the first timefalls within a temperature range specified in the temperaturerequirement; and in response to the temperature of the first product atthe first time falling outside of the temperature range: initializing anelectronic notification; identifying the first product in the electronicnotification; identifying the enclosed cooling unit in the electronicnotification; writing a prompt to remove the first product from theenclosed cooling unit to the electronic notification; and serving theelectronic notification to a computing device affiliated with anassociate of the store.
 12. The method of claim 1, wherein identifyingthe set of products stocked in the cooling unit comprises: detecting afirst shelf of the cooling unit in a first region of the color image;identifying an address of the first shelf; based on the address of thefirst shelf, retrieving a first set of template images from a databaseof template images, each template image in the first set of templateimages comprising visual features of a product in the first list ofproducts assigned to the first shelf by a planogram of the store;extracting a first set of features from the first region of the colorimage; and identifying presence of a first subset of products, in thefirst list of products, on the first shelf based on alignment of thefirst set of features and template images in the first set of templateimages.
 13. The method of claim 12, further comprising: detectingabsence of a first product, in the first list of products, on the firstshelf in response to absence of features, in the first set of features,that align with template images, in the first set of template images,depicting the first product; in response to detecting absence of thefirst product on the first shelf, generating a restocking prompt torestock the first shelf with units of the first product; and serving therestocking prompt to a computing device affiliated with an associate ofthe store.
 14. The method of claim 1, further comprising: at the roboticsystem, during the first scan routine: autonomously navigating toward aheating unit in the store; recording a second color image of the heatingunit; and scanning a second set of temperatures within the heating unit;identifying a second set of products stocked in the heating unit basedon features detected in the second color image; mapping the second setof temperatures to the second set of products at a second time duringthe first scan routine based on positions of products in the second setof products identified in the second color image; and appending therecord with temperatures of the second set of products stocked in theheating unit at the second time.
 15. The method of claim 1, furthercomprising: calculating a mean temperature of the cooling unit at thefirst time based on the set of temperatures within the cooling unitduring the first scan routine; accessing a current temperature settingof the cooling unit; and in response to the mean temperature of thecooling unit deviating from the current temperature setting by more thana threshold difference, serving a maintenance prompt to inspect thecooling unit to a computing device affiliated with an associate of thestore.
 16. The method of claim 1, further comprising: at the roboticsystem, during a second scan routine succeeding the first scan routine:autonomously navigating toward the cooling unit in the store; recordinga second color image of the cooling unit; and scanning a second set oftemperatures within the cooling unit; identifying the set of productsstocked in the cooling unit at a second time during the second scanroutine based on features detected in the second color image; mappingthe second set of temperatures to the set of products based on positionsof products in the set of products identified in the second color image;and estimating temperature exposures of products in the set of productsbetween the first time and the second time by integrating between thefirst set of temperatures and the second set of temperatures of the setof products based on a time duration between the first time and thesecond time.
 17. The method of claim 16, further comprising: accessing ashelf life model of a particular product in the set of products;estimating a remaining shelf life of the particular product based on theshelf life model and a temperature exposure of the particular product;and in response to the remaining shelf life of the particular productfalling below a threshold duration: generating an electronicnotification identifying the particular product and the remaining shelflife of the particular product; and serving the electronic notificationto a computing device affiliated with an associate of the store.